Electronic device for displaying image by using camera monitoring system (cms) side display mounted in vehicle, and operation method thereof

ABSTRACT

A method, performed by an electronic device installed in a vehicle, of switching a view of an image displayed on a camera monitoring system (CMS) side display, and an electronic device are provided. The disclosure includes an electronic device for displaying, on a camera monitoring system (CMS) side display, a first image representing a surrounding environment image, detecting a lane change signal of the vehicle, and, in response to the detected lane change signal, switching the first image displayed on the CMS side display to a second image representing a top view image showing locations of the vehicle and a surrounding vehicle in a virtual image as looking down from above the vehicle, and displaying the second image, and displaying a lane change user interface (UI) indicating information about whether a lane change is possible on the second image, and an operation method thereof.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application, claiming priority under§ 365(c), of an International application No. PCT/KR2021/003647, filedon Mar. 24, 2021, which is based on and claims the benefit of a Koreanpatent application number 10-2020-0037058, filed on Mar. 26, 2020, inthe Korean Intellectual Property Office, the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to a method, performed by an electronic deviceinstalled in a vehicle, of displaying a surrounding environment imagecaptured by using an external camera of a camera monitoring system (CMS)on a CMS side display and switching a view of the surroundingenvironment image, and an electronic device.

2. Description of Related Art

Side view mirrors disposed on left and right sides of a vehicle eachinclude a mirror and a housing and are used for changing lanes orchecking surrounding situations such as surrounding vehicles andpedestrians. With the recent developments in the field of electronicdevices installed in vehicles, the existing side view mirrors have beenreplaced with a camera monitoring system (CMS) including an externalcamera and a display.

The recently used CMS provides a user experience similar to that of theside view mirror including the existing mirror by displaying asurrounding environment image captured by using the external camera on aCMS side display as it is. In particular, because the CMS side displaydisplays a surrounding environment image captured through a specificfield of view (FoV), there is a limit to the FoV that a driver is unableto check with respect to a blind spot which may not be captured by theexternal camera, and a problem may arise as to the driver's judgment ina situation that may occur while driving, such as a lane change, a turn,and the like. Accordingly, a conscious and additional action is requiredfor the driver to directly visually check an external environmentoutside a vehicle window.

The existing blind spot monitoring system (BSMS) of the related artmerely provides a presence/absence notification function regardingwhether there is a surrounding vehicle in a blind spot that may not bechecked through a driver's FoV, and is limited for the driver to make adetailed driving judgement in a specific situation. For safe driving, itis necessary to clearly visually provide during driving informationabout changes in the surrounding environment to the driver.

In addition, the existing CMS side display requires a touch input ormanipulation of a related button in order to determine information aboutthe surrounding environment, but a touch or button manipulation actionwhile driving may be a risk factor.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providean electronic device that detects a surrounding environment from asurrounding environment image captured through an external cameraincluded in a camera monitoring system (CMS), switches a view of thesurrounding environment image displayed on a CMS side display based on achange in the surrounding environment in a situation such as changinglanes or entering an intersection, obtains information about thesurrounding environment, and displays a user interface (UI) related to alane change, and an operation method thereof.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, a method, performed byan electronic device installed in a vehicle, of displaying an image byusing a camera monitoring system (CMS) of the vehicle is provided. Themethod includes displaying, on a CMS side display, a first image that isa surrounding environment image obtained by capturing a left surroundingenvironment or a right surrounding environment of the vehicle by usingan external camera disposed on a side surface of the vehicle, detectinga lane change signal of the vehicle, in response to the detected lanechange signal, switching the first image displayed on the CMS sidedisplay to a second image that is a top view image showing locations ofthe vehicle and a surrounding vehicle in a virtual image as looking downfrom above the vehicle, and displaying the second image, and displayinga lane change user interface (UI) indicating information about whether alane change is possible on the second image.

The detecting of the lane change signal may include detecting a userinput for lighting a turn signal, by manipulating a turn signal lever ofthe vehicle.

The detecting of the lane change signal may include detecting a turn orlane change based on driving route information of a navigation system ofthe vehicle.

The displaying of the lane change UI indicating information aboutwhether the lane change is possible may include displaying a UI relatedto driving environment information comprising at least one of a lane, alocation of a surrounding vehicle, a relative speed between the vehicleand the surrounding vehicle, a distance between the vehicle and thesurrounding vehicle, or an expected entry route of each of the vehicleand the surrounding vehicle detected from the surrounding environmentimage.

The displaying of the second image may include overlaying and displayingthe surrounding environment image obtained through a CMS view camera onthe top view image.

The method may further include detecting a lane departure that thevehicle departs from a first lane on which the vehicle is currentlydriving and enters a second lane which is a lane to be entered by apreset range, in response to the detected lane departure, switching thesecond image displayed on the CMS side display to a third imagecomprising only a top view image excluding the overlaid surroundingenvironment image, and displaying the third image on the CMS sidedisplay.

The method may further include obtaining an occupant image by using acamera mounted inside the vehicle, obtaining location information aboutan occupant comprising at least one of a head location, a head rotationdirection, or a gaze direction of the occupant from the obtainedoccupant image, measuring a distance between the occupant and the CMSside display based on the obtained location information about theoccupant, and zooming in or out a field of view (FoV) of the secondimage based on the measured distance.

The method may further include detecting a hand gesture input includingat least one of a pinch in or out or a palm swipe of a vehicle occupantby using a hand gesture sensor mounted inside the vehicle, and zoomingin or out the FoV of the second image based on the detected hand gestureinput.

The displaying of the second image may include displaying the secondimage on the CMS side display and a center information display (CID) ofthe vehicle.

The method may further include detecting a direction of a turn signallighted by a driver input, and the displaying of the second image mayinclude displaying the second image on the CID based on the detecteddirection of the turn signal.

In accordance with another aspect of the disclosure, an electronicdevice configured to display an image using a camera monitoring system(CMS) of a vehicle including an external camera disposed on each of aleft side surface and right side surface of an outside of the vehicle,and configured to capture a surrounding environment of the vehicle andto obtain a first image that is a surrounding environment image, a CMSside display disposed inside the vehicle and displaying the surroundingenvironment image, a memory storing a program comprising one or moreinstructions controlling the electronic device, and a processorconfigured to execute the one or more instructions of the program storedin the memory, detect a lane change signal of the vehicle, in responseto the detected lane change signal, switch the first image displayed onthe CMS side display to a second image that is a top view image showinglocations of the vehicle and a surrounding vehicle in a virtual image aslooking down from above the vehicle, and control the CMS side display todisplay the second image, and display a lane change user interface (UI)indicating information about whether a lane change is possible on thesecond image.

The processor may detect a user input for lighting a turn signal, bymanipulating a turn signal lever of the vehicle, and switch the firstimage displayed on the CMS side display to the second image based on theuser input.

The processor may display a UI related to driving environmentinformation comprising at least one of a lane, a location of asurrounding vehicle, a relative speed between the vehicle and thesurrounding vehicle, a distance between the vehicle and the surroundingvehicle, or an expected entry route of each of the vehicle and thesurrounding vehicle detected from the surrounding environment image onthe CMS side display.

The processor may control the CMS side display to overlay and displaythe surrounding environment image obtained through the external cameraon the top view image.

The processor may detect a lane departure that the vehicle departs froma first lane on which the vehicle is currently driving and enters asecond lane which is a lane to be entered by a preset range, in responseto the detected lane departure, switch the second image displayed on theCMS side display to a third image comprising only a top view imageexcluding the overlaid surrounding environment image, and control theCMS side display to display the third image.

The electronic device may further include an internal camera mountedinside the vehicle and configured to obtain an occupant image bycapturing an occupant, and the processor may to obtain locationinformation about the occupant comprising at least one of a headlocation, a head rotation direction, or a gaze direction of the occupantfrom the occupant image, measure a distance between a driver and the CMSside display based on the obtained location information about theoccupant, and zoom in or out a field of view (FoV) of the second imagebased on the measured distance.

The electronic device may further include a hand gesture sensor mountedinside the vehicle and configured to detect at least one of a pinch inor out or a palm swipe of a vehicle occupant, and the processor maydetect a hand gesture input of the vehicle occupant through the handgesture sensor and zoom in or out the FoV of the second image based onthe detected hand gesture input.

The electronic device may further include a center information display(CID) disposed on a dashboard of the vehicle, and the processor may todisplay the second image on the CMS side display and the CID.

The processor may detect a direction of a turn signal lighted by adriver input, and control the CID to display the second image on the CIDbased on the detected direction of the turn signal.

Another embodiment of the disclosure provides a computer-readablerecording medium recording thereon a program for execution on acomputer.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a conceptual diagram illustrating an operation of anelectronic device including a camera monitoring system (CMS) installedin a vehicle according to an embodiment of the disclosure;

FIG. 2 is a block diagram illustrating a configuration of an electronicdevice according to an embodiment of the disclosure;

FIG. 3 is a block diagram illustrating a configuration of a vehiclesensor module according to an embodiment of the disclosure;

FIG. 4 is a diagram illustrating each component of a CMS installed in avehicle according to an embodiment of the disclosure;

FIGS. 5A, 5B, 5C, and 5D are diagrams illustrating an embodiment of CMSside displays according to various embodiments of the disclosure;

FIGS. 6A and 6B are diagrams illustrating an embodiment in which anelectronic device of the disclosure switches a view of an imagedisplayed on a CMS side display when detecting a lane change signal of avehicle according to various embodiments of the disclosure;

FIGS. 7A, 7B, and 7C are diagrams illustrating an embodiment of an imageand lane change user interfaces (UIs) displayed on a CMS side displayaccording to various embodiments of the disclosure;

FIG. 8 is a flowchart illustrating an operating method of an electronicdevice according to an embodiment of the disclosure;

FIG. 9 is a flowchart illustrating an operating method of an electronicdevice according to an embodiment of the disclosure;

FIG. 10 is a diagram illustrating an embodiment in which an electronicdevice of the disclosure detects a situation including a lane changesignal and a lane departure of a vehicle and switches a view of an imagedisplayed on a CMS side display based on the detected situationaccording to an embodiment of the disclosure;

FIG. 11A is a diagram illustrating an embodiment in which an electronicdevice of the disclosure changes a field of view (FoV) of an imagedisplayed on a CMS side display based on a distance change between theCMS side display and a driver according to an embodiment of thedisclosure;

FIG. 11B is a diagram illustrating an embodiment in which an electronicdevice of the disclosure changes a FoV of an image displayed on a CMSside display based on a distance change between the CMS side display anda driver according to an embodiment of the disclosure;

FIG. 12 is a diagram illustrating an embodiment in which an electronicdevice of the disclosure changes a FoV of an image displayed on a CMSside display based on a hand gesture of an occupant in a vehicleaccording to an embodiment of the disclosure;

FIG. 13 is a diagram illustrating an embodiment in which an electronicdevice of the disclosure displays a top view image on a centerinformation display (CID) according to an embodiment of the disclosure;

FIG. 14A is a diagram illustrating an embodiment in which an electronicdevice of the disclosure displays a top view image on a CMS side displayand a CID in the case of a left-hand drive (LHD) vehicle according to anembodiment of the disclosure; and

FIG. 14B is a diagram illustrating an embodiment in which an electronicdevice of the disclosure displays a top view image on a CMS side displayand a CID in the case of a right-hand drive (RHD) vehicle according toan embodiment of the disclosure.

Throughout the drawings, it should be noted that like reference numbersare used to depict the same or similar elements, features, andstructures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

All terms used herein including technical or scientific terms have thesame meaning as those generally understood by those of ordinary skill inthe art to which the specification belongs.

Throughout the disclosure, when a certain part “includes” a certaincomponent, this indicates that the part may further include anothercomponent instead of excluding another component unless there isdifferent disclosure. In addition, terms such as “. . . unit” and “. . .module” used in the specification refer to units that perform at leastone function or operation, and the units may be implemented as hardwareor software or as a combination of hardware and software.

The expression “configured to” used in the specification may beexchanged with, for example, “suitable for”, “having the capacity to”,“designed to”, “adapted to”, “made to”, or “capable of” in accordancewith circumstances. The term “configured to” does not necessarilyindicate only “specifically designed to” in terms of hardware. Instead,in a certain circumstance, the expression “a system configured to” mayindicate the system “capable of” together with another device orcomponents. For example, “a processor configured to perform A, B, and C”may indicate an exclusive processor (e.g., an embedded processor)configured to perform a corresponding operation or a generic-purposeprocessor (e.g., a central processing unit (CPU) or an applicationprocessor) capable of performing corresponding operations by executingone or more software programs stored in a memory.

FIG. 1 is a conceptual diagram illustrating an operation of anelectronic device including a camera monitoring system (CMS) installedin a vehicle according to an embodiment of the disclosure.

Referring to FIG. 1 , an electronic device 1000 of FIG. 2 may include anexternal camera 1110 disposed outside of the vehicle, CMS side displays1310L and 1310R, and a center information display (CID) 1320. In anembodiment, the external camera 1110 and the CMS side display 1310L and1310R may constitute the CMS.

However, components included in the electronic device 1000 of FIG. 2 arenot limited to those illustrated in FIG. 1 . The components of theelectronic device 1000 will be described in detail with reference toFIG. 2 .

The external camera 1110 may be disposed on each of left and right sidesof the outside of the vehicle. The external camera 1110 may obtainsurrounding environment images by capturing surrounding environments ofthe left, right, and rear of the vehicle in real time. The externalcamera 1110 may capture a moving object including at least one of asurrounding vehicle, a two-wheeled vehicle, or a pedestrian located inthe surrounding environments of the left, right, and rear of thevehicle.

The CMS side displays 1310L and 1310R may display a surroundingenvironment image 100 captured through the external camera 1110. The CMSside displays 1310L and 1310R may include the left CMS side display1310L disposed on the left side of a steering wheel and the right CMSside display 1310R disposed on the right side of the steering wheel anddisposed adjacent to a passenger seat. However, the CMS side displays1310L and 1310R are not limited to the shapes and arrangements shown inFIG. 1 . Another embodiment of the CMS side display 1310 of thedisclosure will be described in detail with reference to FIGS. 5A, 5B,5C, and 5D.

The electronic device 1000 of FIG. 2 may detect a lane change signal ofthe vehicle, and in response to the detected lane change signal, switchthe surrounding environment image 100 displayed on the CMS side displays1310L and 1310R to a top view image 110 and display the top view image110. The top view image 110 means an image of a view showing thesurrounding environment of the vehicle as looking down from above thevehicle by using a surround view monitoring (SVM) system installed inthe vehicle. In yet another embodiment, the electronic device 1000 mayswitch the surrounding environment image 100 displayed on the CMS sidedisplays 1310L and 1310R to the top view image 110 in response to thelane change signal, reduce the size of the surrounding environment image100, and overlay and display the reduced surrounding environment image100 on the top view image 110.

In yet another embodiment, the electronic device 1000 may detect a userinput for activating (lighting) a left or right turn signal, bymanipulating a turn signal lever 2100, and based on the detected userinput, switch the surrounding environment image 100 displayed on the CMSside display 1310 to the top view image 110 based on the detected userinput, and display the top view image 110. The electronic device 1000may switch a view of an image displayed on one of the left CMS sidedisplay 1310L and the right CMS side display 1310R based on a directionof the lighted turn signal. For example, when a user input for lightingthe left turn signal is received through an input of pressing the turnsignal lever 2100 in a downward direction, the electronic device 1000may switch the surrounding environment image 100 displayed on the leftCMS side display 1310L to the top view image 110, and display the topview image 110. For another example, when a user input for lighting theright turn signal is received by manipulating the turn signal lever 2100in an upward direction, the electronic device 1000 may switch thesurrounding environment image 100 displayed on the right CMS sidedisplay 1310R to the top view image 110, and display the top view image110.

In yet another embodiment, the electronic device 1000 may detect a turnsignal or a lane change signal based on driving route information of anavigation system, may switch the surrounding environment image 100displayed on the CMS side display 1310L or 1310R to the top view image110 based on the detected turn signal or lane change signal, and maydisplay the top view image 110. The driving route information of thenavigation system may be displayed on the CID 1320.

The electronic device 1000 may display a lane change user interface (UI)120 indicating information about whether the lane change is possible onthe top view image 110. In yet another embodiment, the electronic device1000 may obtain driving environment information including at least oneof a lane, a location of a surrounding vehicle, a relative speed betweenown vehicle own vehicle and the surrounding vehicle, a distance betweenown vehicle and the surrounding vehicle, or an expected entry route ofthe surrounding vehicle from the surrounding environment image 100, andmay determine a lane change possibility based on the obtained drivingenvironment information. The electronic device 1000 of FIG. 2 maydisplay the lane change UI 120 determined based on the drivingenvironment information on the CMS side displays 1310L and 1310R. In yetanother embodiment, the lane change UI 120 may overlay and display awarning mark on the surrounding vehicle of own vehicle or display awarning phrase (e.g., ‘rear collision warning’). The lane change UI 120will be described in detail with reference to FIG. 7A.

In yet another embodiment, the electronic device 1000 of FIG. 2 mayoutput a warning sound when an unexpected situation occurs, such as whena surrounding vehicle suddenly increases the speed or when thesurrounding vehicle abruptly changes lanes while driving.

The CMS side display of the related art provides a user experiencesimilar to that of the existing side view mirror configured as a mirror,by displaying only the surrounding environment image 100 captured byusing the external camera 1110. In particular, because the CMS sidedisplay of the related art displays only the surrounding environmentimage 100 captured through a specific field of view (FoV), there was alimit to the FoV that a driver is unable to check with respect to ablind spot which may not be captured by the external camera 1110, andthere was a problem in that the driver has a difficulty in intuitivelyjudging a situation in the situation that may occur while driving, suchas a lane change, a turn, and the like. Accordingly, a conscious andadditional action was required for the driver to directly visually checkthe external environment outside a vehicle window in a situation such asa lane change or a turn, and inconvenience existed.

In addition, the existing blind spot monitoring system (BSMS) of therelated art merely provides a presence/absence notification functionregarding whether there is a surrounding vehicle in a blind spot thatmay not be checked through a driver's FoV, and is limited for the driverto make a detailed driving judgement in a specific situation. For safedriving, it is necessary to clearly and visually provide informationabout a change in the surrounding environment while driving.

The electronic device 1000 of FIG. 2 of the disclosure may provide auser eXperience (UX) differentiated from the CMS side display of therelated art, by detecting a signal indicating that the vehicle intendsto change lanes, switching the surrounding environment image 100displayed on the CMS side displays 1310L and 1310R to the top view image110 based on the lane change signal, displaying the top view image 110,and displaying the lane change UI 120 indicating information about thelane change possibility on the top view image 110. In particular, on thetop view image 110, the electronic device 1000 of the disclosure mayintuitively provide relevant information to the driver in a lane changeor turn situation and enhancing reliability and stability of driving, bydisplaying the lane change UI 120 indicating at least one of a lane, alocation of the surrounding vehicle, a relative speed between ownvehicle and the surrounding vehicle, a distance between own vehicle andthe surrounding vehicle, or an expected entry route of the surroundingvehicle. In addition, when the vehicle is driven in an autonomousdriving mode by using a pre-mounted autonomous driving system, theelectronic device 1000 of FIG. 2 of the disclosure may allow the driverto predict the judgment and driving method of the vehicle, by displayingthe lane change UI 120 including a UI regarding an expected entry routeof own vehicle and surrounding vehicle on the CMS side displays 1310Land 1310R, and accordingly, secure reliability for autonomous driving.

FIG. 2 is a block diagram illustrating a configuration of an electronicdevice according to an embodiment of the disclosure.

In an embodiment of the disclosure, an electronic device 1000 may be adevice installed in a vehicle and controls a CMS.

Referring to FIG. 2 , the electronic device 1000 may include a camera1100, an external sensor 1200, a display 1300, a processor 1400, amemory 1500, a transceiver 1600, and a user input unit 1700. The camera1100, the external sensor 1200, the display 1300, the processor 1400,the memory 1500, the transceiver 1600, and the user input unit 1700 maybe electrically and/or physically connected to each other. Componentsillustrated in FIG. 2 are only according to another embodiment of thedisclosure, and the components included in the electronic device 1000are not limited to those illustrated in FIG. 2 . The electronic device1000 may not include some of the components illustrated in FIG. 2 , andmay further include components which are not illustrated in FIG. 2 .

The camera 1100 may include an external camera 1110 and an internalcamera 1120. The camera 1100 may include an image sensor, such as acomplementary metal-oxide semiconductor (CMOS), charge-coupled device(CCD), or active pixel sensor, and a lens such as at least one of alinear lens, a concave lens, a convex lens, a wide angle lens, or a fisheye lens. The camera 1100 may be analog or digital. In yet anotherembodiment, the camera 1100 may include an infrared illumination outputdevice.

The external camera 1110 may be disposed on left and right sides of theoutside of the vehicle. However, the disclosure is not limited thereto,and the external camera 1110 may be disposed not only on the left andright sides of the vehicle, but also on the front and rear surfaces ofthe vehicle. An arrangement of the external camera 1110 will bedescribed in detail with reference to FIG. 4 .

The external camera 1110 may be configured as a CMS view camera thatcaptures a moving object including at least one of a surroundingvehicle, a two-wheeled vehicle, or a pedestrian located in a surroundingenvironment on the left and right sides and rear of the vehicle. Theexternal camera 1110 may obtain a surrounding environment image, bycapturing a surrounding environment image in real time, and may providethe obtained surrounding environment image to the processor 1400.

In yet another embodiment, the external camera 1110 may be configured asa fish eye lens camera. The fish eye lens camera means a camera equippedwith a wide-angle lens of a capturing angle equal to or greater than180°. Because the surrounding environment image captured by using thefish eye lens camera may have a distortion, the processor 1400 maycorrect the surrounding environment image obtained from the externalcamera 1110 to generate a wide image, and may control the surroundingenvironment image according to a steering angle of the vehicle or thespeed of the vehicle to display the surrounding environment image on theCMS side display 1310. The external camera 1110 may be configured as thefish eye lens camera, thereby capturing a blind spot surrounding thevehicle and obtaining a surrounding environment image regarding theblind spot. The ‘blind spot’ means at least one region that a driver isunable to see because a driver's field of view is blocked by a structureof the vehicle. However, the external camera 1110 is not limited to thefish eye lens camera.

The internal camera 1120 may capture an occupant in the vehicle andobtain an image of the occupant in real time. In an embodiment, theinternal camera 1120 may capture a driver's face or a passenger's facein a passenger seat, obtain an occupant image regarding the driver'sface or the passenger's face, and provide the obtained occupant image tothe processor 1400.

In yet another embodiment, the internal camera 1120 may be disposed on aspecific region of the vehicle, for example, on an upper end of adashboard, a room mirror, or a cluster. However, a location at which theinternal camera 1120 is disposed is not limited to the above-describedexample.

In yet another embodiment, the internal camera 1120 may include an eyetracking sensor that captures occupant's pupils including the driver orthe passenger in the passenger seat, and detects a movement of thepupils. In order for the internal camera 1120 to track the location andgaze of the occupant's pupils, an image analysis method or a contactlens method may be used. The image analysis method is an analysistechnology that detects the movement of the occupant's pupils throughanalysis of the occupant image obtained in real time, and calculates adirection of the gaze with respect to a fixed location reflected on thecornea, and the contact lens method is an analysis technology that useslight reflected by a contact lens with an embedded mirror or a magneticfield of a contact lens embedded in a coil. The internal camera 1120 mayobtain the occupant image and sense the direction of the gaze of theoccupant. However, the technology used by the internal camera 1120 ofthe disclosure to sense the direction of the gaze of the occupant is notlimited to the above-described technologies. In an embodiment, theinternal camera 1120 may include an infrared (IR) sensor, and may usethe IR sensor to track the location of the occupant's pupils and sensethe direction of the gaze.

The external sensor 1200 may include a radar sensor 1210 and anultrasonic sensor 1220. The radar sensor 1210 and the ultrasonic sensor1220 may be disposed on the front and rear surfaces of the vehicle,respectively. The arrangement of the radar sensor 1210 and theultrasonic sensor 1220 will be described in detail with reference toFIG. 4 .

The radar sensor 1210 may generate an electromagnetic wave in a radiowave or microwave spectrum in order to detect a moving object around thevehicle, for example, a surrounding vehicle, a two-wheeled vehicle, apedestrian, etc., and may include a transmission antenna that radiatesthe electromagnetic wave and a reception antenna that receives theelectromagnetic wave. The transmission antenna may radiate the radiowave (pulsed or continuous), receive the radio wave reflected from themoving object by using the reception antenna, and provide informationabout at least one of the location, speed, or angle of the moving objectto the processor 1400. The radar sensor 1210 may communicate with theprocessor 1400 using a wired, wireless, or waveguide method.

In yet another embodiment, the radar sensor 1210 may include a lidarthat uses ultraviolet, visible, or near-infrared light of a laser.

The ultrasonic sensor 1220 may include at least one transducer thatconverts an electric signal into an ultrasonic signal and converts anultrasonic echo signal reflected from the moving object into an electricsignal. The ultrasonic sensor 1220 may transmit an ultrasonic signal toa moving object, including a surrounding vehicle, a two-wheeled vehicle,or a pedestrian, located around the vehicle, and may receive anultrasonic echo signal reflected from the moving object. The ultrasonicsensor 1220 may provide the received ultrasonic echo signal to theprocessor 1400. The processor 1400 may obtain information about alocation of the moving object and a distance between the moving objectand own vehicle, by analyzing the ultrasonic echo signal. In anembodiment, the processor 1400 may use a method of measuring a timeinterval between a time at which an ultrasound wave is transmitted and atime at which the ultrasound echo signal is received, in order tomeasure the distance to the moving object, but is not limited thereto.

The display 1300 may include the CMS side display 1310 and the CID 1320.The display 1300 may be configured as a physical device including atleast one of, for example, a liquid crystal display (LCD) display, aplasma display panel (PDP) display, an organic light emitting diode(OLED) display, a field emission display (FED), a light emitting diode(LED) display, a vacuum fluorescent display (VFD), a digital lightprocessing (DLP) display, a flat panel display, a 3D display, or atransparent display, but is not limited thereto. In yet anotherembodiment, the display 1300 may be configured as a touch screenincluding a touch panel 1710 that detects a touch input of a user (e.g.,a driver or a passenger).

The CMS side display 1310 may display a surrounding environment imagecaptured through the external camera 1110. In yet another embodiment,the CMS side display 1310 may display a top view image which is an imageshowing the surrounding environment of the vehicle as looking down fromabove the vehicle by using a SVM system installed in the vehicle. Thetop view image may be a surround view image showing locations of ownvehicle and the surrounding vehicle in a virtual image.

The external camera 1110 and the CMS side display 1310 may constitute aCMS. However, the disclosure is not limited thereto, and the CMS mayinclude at least one component of the electronic device 1000.

The CMS side display 1310 may include the left CMS side display 1310L(see FIG. 1 ) disposed on the left side of a steering wheel and theright CMS side display 1310R (see FIG. 1 ) disposed on the right side ofthe steering wheel and disposed adjacent to the passenger seat, but isnot limited thereto. In yet another embodiment, the CMS side display1310 may be included in a cluster display of the vehicle, or may becombined with the CID 1320. Another embodiment of the CMS side display1310 of the disclosure will be described in detail with reference toFIGS. 5A to 5D.

The CID 1320 may display a direction navigation to a destination ordisplay vehicle-related information. In yet another embodiment, the CID1320 may display image content, such as a movie, a game, etc. The CID1320 may be disposed between a driver seat and a passenger seat on adashboard of the vehicle.

In yet another embodiment, the CID 1320 may display the top view imageunder the control of the processor 1400. An embodiment in which the topview image is displayed on the CID 1320 will be described in detail withreference to FIGS. 13, 14A, and 14B.

The processor 1400 may execute one or more instructions of a programstored in the memory 1500. The processor 1400 may be configured as ahardware component that performs arithmetic, logic, input/outputoperations and signal processing. The processor 1400 may be configuredas at least one of, for example, a central processing unit, amicroprocessor, a graphic processing unit, Application SpecificIntegrated Circuits (ASICs), Digital Signal Processors (DSPs), DigitalSignal Processors (DSPDs), Signal Processing Devices, Programmable LogicDevices (PLDs), or Field Programmable Gate Arrays (FPGAs), but is notlimited thereto.

The memory 1500 may include, for example, a flash memory type memory, ahard disk type memory, a multimedia card micro type memory, a card typememory (e.g., SD or XD memory, etc.), non-volatile memory including atleast one of read-only memory (ROM), electrically erasable programmableread-only memory (EEPROM), or programmable read-only memory (PROM), andvolatile memory such as random access memory (RANI) or static randomaccess memory (SRAM).

The memory 1500 may store instructions, data structures, and programcodes readable by the processor 1400. In the following embodiment, theprocessor 1400 may be implemented by executing instructions or codes ofa program stored in the memory 1500. For example, program command codesthat detect a lane change signal of the vehicle, in response to thedetected lane change signal, switch a surrounding environment imagedisplayed on the CMS side display 1310 to a top view image, display thetop view image, and display a lane change UI indicating informationabout whether a lane change is possible on the top view image may bestored in the memory 1500.

In yet another embodiment, the processor 1400 may use a configurationimplemented as hardware and software included in at least one of a BSMS,a SVM, or a lane departure warning system (LDWS) installed in thevehicle.

In yet another embodiment, the processor 1400 and the memory 1500 mayconstitute a control unit.

The processor 1400 may detect the lane change signal of the vehicle, inresponse to the detected lane change signal, switch the surroundingenvironment image displayed on the CMS side display 1310 to the top viewimage, display the top view image, and display the lane change UIindicating the information about whether the lane change is possible onthe top view image.

In yet another embodiment, the processor 1400 may detect a user inputfor activating (lighting) a turn signal, by manipulating the turn signallever 2100 (see FIG. 1 ) of the vehicle, and based on the detected userinput, and switch the surrounding environment image displayed on the CMSside display 1310 to the top view image based on the user input. Theprocessor 1400 may obtain activation (lighting) information of the turnsignal according to a user input for manipulating the turn signal lever2100 from a vehicle sensor module 2000 through the transceiver 1600. Inan embodiment, the processor 1400 may switch a view of an imagedisplayed on any one of the CMS side display 1310L (see FIG. 1 )disposed on the left and the CMS side display 1310R (see FIG. 1 )disposed on the right based on a direction of the activated turn signal.For example, when a user input for lighting the left turn signal isreceived through an input of pressing the turn signal lever 2100 in adownward direction, the processor 1400 may receive activationinformation of the left turn signal from the transceiver 1600, switchthe surrounding environment image displayed on the left CMS side display1310L to the top view image, and display the top view image. For anotherexample, when a user input for lighting the right turn signal isreceived by manipulating the turn signal lever 2100 in an upwarddirection, the processor 1400 may receive activation information of theright turn signal from the transceiver 1600, may switch the surroundingenvironment image displayed on the right CMS side display 1310R to thetop view image, and may display the top view image.

In yet another embodiment, the processor 1400 may detect a turn signalor a lane change signal based on route information of a navigationsystem installed in the vehicle, switch the surrounding environmentimage displayed on the CMS side display 1310 to the top view image basedon the detected turn signal or lane change signal, and display the topview image. When the vehicle is driven in an autonomous driving mode,the processor 1400 may obtain driving information such as a steeringangle, a speed, and the like, from the vehicle sensor module 2000, andswitch the view of the image displayed on the CMS side display 1310based on the obtained driving information and the route information ofthe navigation system. For example, the processor 1400 may detect asituation in which the vehicle turns at a crossroad using the routeinformation of the navigation system, and in a case in which changeinformation of the steering angle is obtained from the vehicle sensormodule 2000 in the situation, switch a surrounding environment imagedisplayed on the CMS side display 1310 to a top view image.

In yet another embodiment, the processor 1400 may switch the surroundingenvironment image displayed on the CMS side display 1310 to the top viewimage in response to the lane change signal, reduce a frame size of thesurrounding environment image, and overlay the surrounding environmentimage on the top view image. The processor 1400 may control the CMS sidedisplay 1310 to display the surrounding environment image overlaid onthe top view image.

In yet another embodiment, the processor 1400 may use the surroundingenvironment image to obtain driving environment information including atleast one of a lane, locations of own vehicle and the surroundingvehicle, a relative speed between own vehicle and the surroundingvehicle, a distance between own vehicle and the surrounding vehicle, oran expected entry route of the surrounding vehicle. The processor 1400may detect the lane and the surrounding vehicle from the surroundingenvironment image, and may obtain location information about thesurrounding vehicle, by analyzing the surrounding environment imageobtained through the external camera 1110. In yet another embodiment,the processor 1400 may detect the surrounding vehicle moving around ownvehicle, by analyzing a plurality of image frames included in asurrounding environment image obtained from the external camera 1110 inreal time. The processor 1400 may detect the surrounding vehicle fromthe surrounding environment image using, for example, image processingtechnology or machine learning including a deep neural network. Theprocessor 1400 may track the location of the surrounding vehicledetected from the surrounding environment image and update locationinformation about the surrounding vehicle in real time. In yet anotherembodiment, the processor 1400 may predict the expected entry route ofthe surrounding vehicle using the location information about thesurrounding vehicle detected in each of the plurality of frames includedin the surrounding environment image.

In yet another embodiment, the processor 1400 may obtain informationabout the location of the surrounding vehicle, the distance between ownvehicle and the surrounding vehicle, and the relative speed between ownvehicle and the surrounding vehicle, by tracking the location of thesurrounding vehicle using not only the surrounding environment image butalso the radar sensor 1210 and the ultrasonic sensor 1220. A method,performed by the processor 1400, of obtaining information such as thelocation, the speed, and the expected entry route of the surroundingvehicle by using the surrounding environment image and the radar sensor1210 and the ultrasonic sensor 1220 is an example and is not limitedthereto.

The processor 1400 may determine a lane change possibility with respectto a lane to be entered based on the driving environment informationincluding at least one of the obtained locations of own vehicle and thesurrounding vehicle, relative speed between own vehicle and thesurrounding vehicle, distance between own vehicle and the surroundingvehicle, or expected entry route of the surrounding vehicle. Theprocessor 1400 may display the lane change UI indicating the informationabout whether the lane change is possible on the top view imagedisplayed on the CMS side display 1310. A specific embodiment of thelane change UI will be described in detail with reference to FIGS. 7A,7B, and 7C.

In yet another embodiment, the processor 1400 may detect a lanedeparture that the vehicle departs from a first lane on which thevehicle is currently driving and enters a second lane which is the laneto be entered by a preset range, and in response to the detected lanedeparture, may control the CMS side display 1310 to display only the topview image on the CMS side display 1310. When the top view image and thesurrounding environment image overlaid on the top view image aredisplayed on the CMS side display 1310 together, the lane departure isdetected, and therefore, the processor 1400 may control the CMS sidedisplay 1310 to display only the top view image excluding thesurrounding environment image. An embodiment in which the lane departureis detected, and therefore, the view of the image displayed on the CMSside display 1310 is switched will be described in detail with referenceto FIG. 10 .

In yet another embodiment, the processor 1400 may obtain locationinformation about the occupant including at least one of a head locationof the occupant, a head rotation direction, or a gaze direction from theoccupant image obtained through the internal camera 1120, measure adistance between the occupant and the CMS side display 1310 based on thelocation information about the occupant, and zoom in or out a FoV of thetop view image based on the measured distance. In yet anotherembodiment, the processor 1400 may recognize a driver's face or apassenger's face in the passenger seat from the occupant image, and mayextract main feature points, such as eyes, nose, mouth, and the like,from the recognized face. In this case, the processor 1400 may use awell-known image processing technology or a machine learning-based imageanalysis technology using a deep neural network, but is not limitedthereto. The processor 1400 may obtain three-dimensional (3D) locationcoordinate values of the extracted main feature points, and may obtainthe occupant location information including at least one of the headlocation of the occupant, the head rotation direction, or the gazedirection using the obtained the 3D location coordinate values. Theprocessor 1400 may measure a distance between the occupant's face andthe CMS side display 1310 using the 3D location coordinate values of themain feature points extracted from the occupant image and a locationcoordinate value of the CMS side display 1310. The processor 1400 mayzoom in or out the FoV of the top view image displayed on the CMS sidedisplay 1310 based on the measured distance. Another embodiment in whichthe processor 1400 adjusts the FoV of the top view image based on thedistance between the occupant's face and the CMS side display 1310 willbe described in detail with reference to FIGS. 11A and 11B.

In yet another embodiment, the processor 1400 may receive a hand gestureincluding at least one of a pinch in or out or a palm swipe of theoccupant from a hand gesture input unit 1720 and zoom in or out the FoVof the top view image displayed on the CMS side display 1310 based onthe hand gesture. Another embodiment in which the hand gesture isdetected, and therefore, the FoV of the top view image displayed on theCMS side display 1310 is adjusted will be described in detail withreference to FIG. 12 .

In yet another embodiment, the processor 1400 may detect a direction ofa turn signal lighted by a driver input, and display the top view imageon the CID 1320 based on the detected direction of the turn signal.Another embodiment in which the top view image is displayed on the CID1320 will be described in detail with reference to FIGS. 14A and 14B.

The transceiver 1600 may perform data communication between theelectronic device 1000 and the vehicle sensor module 2000 mounted on thevehicle. In an embodiment, the electronic device 1000 may communicatewith the vehicle sensor module 2000 according to a controller areanetwork (CAN).

In yet another embodiment, the transceiver 1600 may receive at least oneof activation of the turn signal, information about a direction of theactivated turn signal, speed sensing information, steering angleinformation of the vehicle, pedal sensing information, or gear leversensing information from the vehicle sensor module 2000, by performingCAN communication. The vehicle sensor module 2000 will be described indetail with reference to FIG. 3 .

The user input unit 1700 may include the touch panel 1710 and the handgesture input unit 1720.

The touch panel 1710 may be combined with the display 1300 to provide atouch screen. For example, the touch screen may include an integratedmodule in which the touch panel 1710 is combined with the CID 1320 in astack structure. For another example, the touch panel 1710 may becombined with the CMS side display 1310 in the stack structure, and theCMS side display 1310 may be implemented as the touch screen. When theCMS side display 1310 or the CID 1320 is implemented as the touchscreen, the touch panel 1710 may receive a user touch input and displaya graphical user interface (GUI).

The touch panel 1710 may sense the user touch input and output a touchevent value corresponding to the sensed touch input. The touch panel1710 may be implemented as various types of touch sensors, such as acapacitive type touch sensor, a pressure sensitive type touch sensor, apiezoelectric type touch sensor, and the like.

The hand gesture input unit 1720 may detect at least one of the pinch inor out or a palm swipe of the vehicle occupant.

FIG. 3 is a block diagram illustrating a configuration of the vehiclesensor module according to an embodiment of the disclosure.

Referring to FIG. 3 , a vehicle sensor module 2000 may be mounted on avehicle, sense a manipulation of a turn signal, a speed, a steeringangle, a pedal, and a gear lever of the vehicle, and the like, obtaininformation related to driving of the vehicle, and provide informationrelated to driving to the electronic device 1000. The vehicle sensormodule 2000 may transmit the information related to driving of thevehicle to the electronic device 1000 through the transceiver 1600 ofthe electronic device 1000. The vehicle sensor module 2000 may performCAN communication with the electronic device 1000.

The vehicle sensor module 2000 is a separate component from theelectronic device 1000. However, the disclosure is not limited thereto,and at least one component included in the vehicle sensor module 2000may be included in the electronic device 1000.

The vehicle sensor module 2000 may include a turn signal sensor 2010, aspeed sensor 2020, a steering angle sensor 2030, a pedal sensor 2040,and a gear lever sensor 2050.

The turn signal sensor 2010 may sense whether the turn signal isactivated by a driver's manipulation. The turn signal sensor 2010 maysense a direction of the lighted the turn signal, and transmitinformation about the sensed lighted direction of the turn signal to theelectronic device 1000. For example, the turn signal sensor 2010 mayobtain information regarding which turn signal is lighted among a leftdirection turn signal and a right direction turn signal.

The speed sensor 2020 may sense a driving speed of the vehicle and maytransmit driving speed information to the electronic device 1000.

The steering angle sensor 2030 may sense a steering angle of the vehicleby manipulation of a steering wheel and may transmit steering angleinformation to the electronic device 1000.

The pedal sensor 2040 may sense a pressure according to a driver'smanipulation of an accelerator pedal or a brake pedal, and may transmitinformation about the pressure applied to the pedal to the electronicdevice 1000.

The gear lever sensor 2050 may sense manipulation of a transmission gearlever by the driver's manipulation, and may transmit information aboutthe manipulation of the transmission gear lever to the electronic device1000.

The electronic device 1000 may detect driving of the vehicle and a lanechange signal, based on at least one of the information about theactivation of the turn signal and the lighting direction, the speedinformation, the steering angle information, the pedal pressureinformation, or the gear lever manipulation information received fromthe vehicle sensor module 2000.

FIG. 4 is a diagram illustrating each component of a CMS installed inthe vehicle according to an embodiment of the disclosure.

Referring to FIG. 4 , the external camera 1110, a front camera 1112, arear camera 1114, the radar sensor 1210, and the ultrasonic sensor 1220may be arranged in the vehicle 10.

The external camera 1110 may be disposed on left and right sides of thevehicle 10. The external camera 1110 may obtain a surroundingenvironment image of a moving object including at least one of asurrounding vehicle, a two-wheeled vehicle, or a pedestrian, bycapturing surrounding environment of the left and right sides and therear of the vehicle 10.

The front camera 1112 may be disposed on a front part of the vehicle 10.In an embodiment, the front camera 1112 may include a lane camera thatcaptures the front of the vehicle 10 to obtain image information of alane.

The rear camera 1114 may be mounted on a rear part of the vehicle 10,for example, a trunk, a rear bumper, a rear pillar, or a rearwindshield. The rear camera 1114 may capture the rear of the vehicle 10to obtain a rear environment image.

In another embodiment, the external camera 1110, the front camera 1112,and the rear camera 1114 may constitute a SVM system. The SVM system isa system that provides an image showing a situation of a surrounding360° space of the vehicle 10 as looking down from above the vehicle 10in real time. The SVM system may correct a distortion of the surroundingenvironment image captured and input from each of the external camera1110, the front camera 1112, and the rear camera 1114, and may generatea top view image as looking down from above in the form of a bird's eyeview of the surrounding of the vehicle 10 using a time change and animage synthesis technology.

The radar sensor 1210 may be disposed on the front part of the vehicle10. For example, the radar sensor 1210 may be mounted on a front grill,a front bumper, or a front lamp of the vehicle 10, but is not limitedthereto. The radar sensor 1210 may generate an electromagnetic wave in aradio wave or microwave spectrum to detect a moving object locatedaround the vehicle 10, for example, a surrounding vehicle, a two-wheeledvehicle, a pedestrian, and the like, and may include a transmissionantenna that radiates the electromagnetic wave and a reception antennathat receives the electromagnetic wave. The transmission antenna mayradiate the radio wave (pulsed or continuous), may receive the radiowave reflected from the moving object by using the reception antenna,and may provide information about at least one of the location, speed,or angle of the moving object to the processor 1400 (see FIG. 2 ).

The ultrasonic sensor 1220 may be disposed on the front part of thevehicle 10. The ultrasonic sensor 1220 may be mounted on, for example, afront bumper of the vehicle 10 or a front lamp, but is not limitedthereto. The ultrasonic sensor 1220 may transmit an ultrasonic signal toa moving object, including a surrounding vehicle, a two-wheeled vehicle,or a pedestrian, located in the front and rear of the vehicle 10, andmay receive an ultrasonic echo signal reflected from the moving object.The ultrasonic sensor 1220 may provide the received ultrasonic echosignal to the processor 1400.

FIGS. 5A, 5B, 5C, and 5D are diagrams illustrating an embodiment of CMSside displays according to various embodiments of the disclosure.

Referring to FIG. 5A, the CMS side displays 1310 a may be disposed onthe left and right sides of a steering wheel inside a vehicle,respectively. The left CMS side display 1310 a may be disposed adjacentto a driver seat. Referring to FIG. 1 together, the right CMS sidedisplay 1310R may be disposed adjacent to a passenger seat. In anembodiment, the CMS side display 1310 a may be mounted on or included ina dashboard, but is not limited thereto. In an embodiment, the CMS sidedisplays 1310 a may be mounted on a door trim of the driver seat and adoor trim of the passenger seat, respectively.

Referring to FIG. 5B, the CMS side display 1310 a may be included in acluster display 1330.

The cluster display 1330 may display vehicle information including atleast one of a driving state of the vehicle or operating stateinformation of various devices. The cluster display 1330 may be includedin the dashboard. The cluster display 1330 may display, for example, atleast one of revolutions per minute (RPM) information of an engine,speed information, turn signal activation state information, fuelinformation, or coolant temperature information. The cluster display1330 may include a fuel gauge, a water temperature gauge, an enginethermometer, various warning lamps, etc. indicating the state of thevehicle, in addition to a speedometer that displays a driving speed anda mileage of the vehicle and a clock together, a tachometer thatdisplays the RPM information of the engine, and a tripmeter thatdisplays the mileage.

The cluster display 1330 may be configured as a physical deviceincluding at least one of, for example, a liquid crystal display (LCD)display, a plasma display panel (PDP) display, an organic light emittingdiode (OLED) display, a field emission display (FED), a light emittingdiode (LED) display, a vacuum fluorescent display (VFD), a digital lightprocessing (DLP) display, a flat panel display, a 3D display, or atransparent display, but is not limited thereto.

The CMS side display 1310 b may be included in a partial region withinthe cluster display 1330. In an embodiment, the CMS side display 1310 bmay replace a speedometer that displays the driving speed information ofthe cluster display 1330 to display at least one of a surroundingenvironment image or a top view image. In another embodiment, the CMSside display 1310 b may replace the tachometer that displays the RPMinformation of the engine of the cluster display 1330 to display atleast one of the surrounding environment image or the top view image.

In an embodiment, when the processor 1400 (see FIG. 2 ) receivesinformation indicating that a left turn signal is activated from theturn signal sensor (2010, see FIG. 3 ) of the vehicle sensor module 2000(see FIG. 3 ), a region of the cluster display 1330 that displays thespeedometer may be replaced with the CMS side display 1310 b. Similarly,when the processor 1400 receives information indicating that a rightturn signal is activated from the turn signal sensor 2010, a region ofthe cluster display 1330 that displays the tachometer may be replacedwith the CMS side display 1310 b.

Referring to FIG. 5C, the CMS side display 1310 c may be included on apartial region of the cluster display 1330. In an embodiment, the CMSside display 1310 c may display at least one of the surroundingenvironment image or the top view image between the regions of the ofthe cluster display 1330 that display the driving speedometer and thetachometer. The description of the cluster display 1330 is the same asthat of the cluster display 1330 illustrated in FIG. 5B, and thus aredundant description thereof is omitted.

Referring to FIG. 5D, the CMS side display 1310 d may be included in apartial region within the CID 1320. The CID 1320 may be disposed betweenthe driver seat and the passenger seat on the dashboard of the vehicle.The CID 1320 may display a direction navigation to a destination ordisplay a top view image of a surrounding vehicle.

In an embodiment, the CMS side display 1310 d may display at least oneof the surrounding environment image or the top view image on a dividedpartial region within the CID 1320.

FIGS. 6A and 6B are diagrams illustrating an embodiment in which anelectronic device of the disclosure switches a view of an imagedisplayed on the CMS side display when detecting a lane change signal ofa vehicle according to various embodiments of the disclosure.

Referring to FIG. 6A, the CMS side display 1310 may display thesurrounding environment image 100 obtained by in real time capturing asurrounding environment of the left and right sides and the rear of thevehicle captured through the external camera 1110 (see FIGS. 1 and 2 ).

While the CMS side display 1310 displays the surrounding environmentimage 100, the processor 1400 (see FIG. 2 ) of the electronic device1000 may detect the lane change signal of the vehicle. When a drivermanipulates the turn signal lever 2100 (see FIG. 1 ) of the vehicle andactivates (lights) a turn signal, the processor 1400 may detect the lanechange signal of the vehicle, by obtaining activation information of theturn signal. In another embodiment, the processor 1400 may detect thelane change signal including a turn or a lane change based on routeinformation of a navigation system installed in the vehicle.

When detecting the lane change signal of the vehicle, the electronicdevice 1000 may switch the surrounding environment image 100 displayedon the CMS side display 1310 to the top view image 110, and display thetop view image 110 on the CMS side display 1310. The top view image 110is an image obtained using a SVM system installed in the vehicle, anddisplays locations of own vehicle and the surrounding vehicle detectedfrom the surrounding environment image 100 in a virtual image. The topview image 110 may mean an image showing a situation of the surrounding360° space of own vehicle as looking down from the vehicle.

Referring to FIG. 6B, when detecting the lane change signal, theelectronic device 1000 may switch the surrounding environment image 100displayed on the CMS side display 1310 to the top view image 110, reducea frame size of the surrounding environment image 100, and display thereduced surrounding environment image 100 by overlaying the reducedsurrounding environment image 100 on the switched top view image 110.The processor 1400 (see FIG. 2 ) of the electronic device 1000 maycontrol the CMS side display 1310 to display the surrounding environmentimage 100 overlaid on the top view image 110.

The embodiment shown in FIG. 6B is the same as the embodiment shown inFIG. 6A except for a feature of overlaying and displaying thesurrounding environment image 100 on the top view image 110, and thus aredundant description thereof is omitted.

FIGS. 7A, 7B, and 7C are diagrams illustrating an embodiment of the topview image and lane change UIs displayed on a CMS side display accordingto various embodiments of the disclosure.

Referring to FIG. 7A, the lane change UI 120 may be displayed on the topview image 110. The lane change UI 120 may include an own vehicle image121, a first surrounding vehicle image 122, an expected entry route UI123 of a surrounding vehicle, a warning icon 124, and a warning phrase125.

The processor 1400 of the electronic device 1000 may obtain drivingenvironment information including at least one of a lane, locations ofown vehicle and the surrounding vehicle, a relative speed between ownvehicle and the surrounding vehicle, a distance between own vehicle andthe surrounding vehicle, or an expected entry route of the surroundingvehicle, by analyzing the surrounding environment image captured byusing the external camera 1110 (see FIGS. 1 and 2 ). In an embodiment,the processor 1400 may obtain information about the location of thesurrounding vehicle, the distance between own vehicle and thesurrounding vehicle, and the relative speed between own vehicle and thesurrounding vehicle, by tracking the location of the surrounding vehicleusing not only the surrounding environment image but also the radarsensor 1210 (see FIGS. 2 and 4 ) and the ultrasonic sensor 1220 (seeFIGS. 2 and 4 ).

The processor 1400 may generate the own vehicle image 121 and the firstsurrounding vehicle image 122 based on the obtained locations of ownvehicle and the surrounding vehicle, generate the expected entry routeUI 123 indicating an expected entry route of the surrounding vehicle,and display the generated images 121 and 122 and entry route UI 123 onthe top view image 110. The processor 1400 may display a warning UIinforming that a situation such as a collision may occur when changinglanes on the top view image 110, based on the distance between ownvehicle and the surrounding vehicle, the relative speed, and theexpected entry route of the surrounding vehicle. The warning UI mayinclude the warning icon 124 and the warning phrase 125.

Referring to FIG. 7B, the lane change UI 130 may be displayed on the topview image 110. The lane change UI 130 may include an own vehicle image131, an entry route UI 132 of own vehicle, a first surrounding vehicleimage 133, an expected entry route UI 134 of a first surroundingvehicle, a second surrounding vehicle image 135, an expected entry routeUI 136 of a second surrounding vehicle, a warning mark 137, and awarning phrase 138.

The embodiment shown in FIG. 7B is the same as the embodiment shown inFIG. 7A except that the lane change UI 130 displayed on the top viewimage 110 has a plurality of surrounding vehicles (the first surroundingvehicle and the second surrounding vehicle), and displays an expectedentry route of each of the plurality of surrounding vehicles as an UI,and thus, a redundant description thereof is omitted. In the embodimentshown in FIG. 7B, the warning mark 137 and the warning phrase 138 thatwarn of the second surrounding vehicle spaced apart from the currentvehicle by two lanes may be displayed on the top view image 110. Thewarning phrase 138 may be, for example, ‘left vehicle warning’.

Referring to FIG. 7C, the lane change UI 140 may be displayed on thesurrounding environment image 100. The lane change UI 140 may include alane changeability notification UI 141 and an acceleration UI 142.

The lane changeability notification UI 141 may display information aboutwhether own vehicle may depart from a lane on which own vehicle iscurrently driving and change to a lane on which the surrounding vehicleis driving, based on the relative speed of own vehicle and thesurrounding vehicle and the location between own vehicle and thesurrounding vehicle. In an embodiment, the processor 1400 of theelectronic device 1000 may analyze the surrounding environment image100, may determine a lane change possibility with respect to a lane tobe entered based on at least one piece of information of the locationsof own vehicle and the surrounding vehicle, the relative speed betweenown vehicle and the surrounding vehicle, the distance between ownvehicle and the surrounding vehicle, or the expected entry route of thesurrounding vehicle obtained by using the radar sensor 1210 (see FIGS. 2and 4 ) and the ultrasonic sensor 1220 (see FIGS. 2 and 4 ), and maygenerate the lane changeability notification UI 141 based on adetermination result. The processor 1400 may display the lanechangeability notification UI 141 on the CMS side display.

The acceleration UI 142 is a UI indicating a speed that needs to beaccelerated in order to change the current lane to a lane that ownvehicle intends to enter. The processor 1400 may measure the speed atwhich own vehicle needs to accelerate in order to change the lane to beentered based on the distance between own vehicle and the surroundingvehicle and the relative speed between own vehicle and the surroundingvehicle, and generate the acceleration UI 142 based on a measurementresult. The processor 1400 may display the acceleration UI 142 on theCMS side display.

In the embodiment illustrated in FIG. 7C, the lane changeabilitynotification UI 141 and the acceleration UI 142 are illustrated as beingdisplayed on the surrounding environment image 100, but are not limitedthereto. In an embodiment, the lane changeability notification UI 141and the acceleration UI 142 may be displayed on the top view image 110(see FIGS. 7A and 7B).

In the embodiments shown in FIGS. 6A to 7C, the CMS side display 1310 ofthe disclosure may display the top view image 110, and display the lanechange UIs 120, 130, and 140 indicating at least one of own vehicle, thelane, the location of the surrounding vehicle, the relative speedbetween own vehicle and the surrounding vehicle, the distance betweenown vehicle and the surrounding vehicle, or the expected entry route ofthe surrounding vehicle, thereby intuitively providing relatedinformation to a driver in the lane change or turn situation.Accordingly, the CMS side display 1310 according to an embodiment of thedisclosure may improve driving reliability and provide a sense ofstability to the driver.

FIG. 8 is a flowchart illustrating a method of operating an electronicdevice according to an embodiment of the disclosure.

Referring to FIG. 8 , in operation S810, the electronic device 1000 mayobtain a surrounding environment image by using an external cameradisposed on the side of a vehicle, and display a first image related tothe obtained surrounding environment image. In an embodiment, theexternal camera may be configured as a CMS view camera that captures amoving object including at least one of a surrounding vehicle, atwo-wheeled vehicle, or a pedestrian located in the surroundingenvironments of the left, right, and rear of the vehicle. The externalcamera may obtain the surrounding environment image by capturing thesurrounding environment image in real time, and may provide the obtainedsurrounding environment image to the processor 1400 (see FIG. 2 ). In anembodiment, the processor 1400 may display the surrounding environmentimage on the CMS side display 1310 (see FIG. 2 ).

In operation S820, the electronic device 1000 detects a lane changesignal of the vehicle.

In another embodiment, the electronic device 1000 may detect a userinput for lighting a turn signal by manipulating a turn signal lever ofthe vehicle. In yet another embodiment, the processor 1400 may obtainactivation (lighting) information of the turn signal according to a userinput for manipulating the turn signal lever 2100 (see FIG. 3 ) from thevehicle sensor module 2000 (see FIG. 3 ), and may detect the user inputfor lighting the turn signal based on the activation information of theturn signal. In yet another embodiment, the processor 1400 may obtaininformation about a direction of the activated turn signal (e.g.,activation of the left turn signal or activation of the right indicator)from the vehicle sensor module 2000.

In yet another embodiment, the electronic device 1000 may detect a turnor a lane change based on driving route information of a navigationsystem of the vehicle. In yet another embodiment, when the vehicle isdriven in an autonomous driving mode, the processor 1400 may obtaindriving information such as a steering angle, a speed, and the like,from the vehicle sensor module 2000, and may detect a turn signal or alane change signal based on the obtained driving information. Forexample, the processor 1400 may detect a situation in which the vehicleturns at a crossroad using route information of the navigation system.

In yet another embodiment, the processor 1400 may obtain informationabout a steering angle change of the vehicle from the vehicle sensormodule 2000 and may detect the turn signal or the lane change signalbased on the obtained information about the steering angle change.

In operation S830, in response to the lane change signal, the electronicdevice 1000 may switch a first image to a second image that is a topview image that displays a surrounding vehicle and own vehicle in avirtual image and display the second image.

In yet another embodiment, when the lane change signal is detected, theelectronic device 1000 may switch the first image that is thesurrounding environment image to the second image that is the top viewimage, and may display the switched second image on the CMS side display1310 (see FIGS. 1 and 2 ). The second image is a top view image showingown vehicle and the surrounding vehicle in the virtual image as lookingdown from above in the form of a bird's eye view. The processor 1400 maycapture a situation of a surrounding 360° space of the vehicle by usinga SVM system installed in the vehicle, correct a distortion of acaptured image, and may generate the top view image using a time changeand an image synthesis technology.

According to yet another embodiment, the electronic device 1000 mayswitch a view of an image displayed on any one of the CMS side display1310L (see FIG. 1 ) disposed on the left and the CMS side display 1310R(see FIG. 1 ) disposed on the right based on the direction of theactivated turn signal. For example, when a user input for lighting aleft turn signal is received through an input of pressing the turnsignal lever 2100 (see FIG. 1 ) in a downward direction, the processor1400 may switch the surrounding environment image displayed on the leftCMS side display 1310L to the top view image, and may display the topview image. For another example, when a user input for lighting a rightturn signal is received by manipulating the turn signal lever 2100 in anupward direction, the processor 1400 may switch the surroundingenvironment image displayed on the right CMS side display 1310R to thetop view image, and may display the top view image.

In an embodiment, the processor 1400 may switch the surroundingenvironment image displayed on the CMS side display 1310 to the top viewimage in response to the lane change signal, may reduce a frame size ofthe surrounding environment image, and may overlay the surroundingenvironment image on the top view image. The processor 1400 may controlthe CMS side display 1310 to display the surrounding environment imageoverlaid on the top view image.

In operation S840, the electronic device 1000 may display a lane changeUI indicating information about whether the lane change is possible onthe second image. In yet another embodiment, the electronic device 1000may obtain driving environment information including at least one of alane, locations of own vehicle and the surrounding vehicle, a relativespeed between own vehicle and the surrounding vehicle, a distancebetween own vehicle and the surrounding vehicle, or an expected entryroute of the surrounding vehicle, by analyzing the surroundingenvironment image. In an embodiment, the processor 1400 may obtaininformation about the location of the surrounding vehicle, the distancebetween own vehicle and the surrounding vehicle, and the relative speedbetween own vehicle and the surrounding vehicle, by tracking thelocation of the surrounding vehicle using not only the surroundingenvironment image but also the radar sensor 1210 (see FIGS. 2 and 4 )and the ultrasonic sensor 1220 (see FIGS. 2 and 4 ). The processor 1400may display the lane change UI indicating the information about whetherthe lane change is possible on the CMS side display 1310 (see FIGS. 1and 2 ) based on the obtained driving environment information.

In yet another embodiment, the lane change UI may be displayed on thetop view image displayed on the CMS side display 1310. However, thedisclosure is not limited thereto, and the lane change UI may bedisplayed on the surrounding environment image. For the lane change UI,reference is made to the descriptions of the lane change UIs 120, 130,and 140 respectively illustrated in FIGS. 7A to 7C.

FIG. 9 is a flowchart illustrating a method of operating an electronicdevice according to an embodiment of the disclosure.

The embodiment shown in FIG. 9 relates to operations additionallyperformed between operations S820 and S840 shown in FIG. 8 . OperationS910 is performed is performed after operation S820 of FIG. 8 isperformed.

Referring to FIG. 9 , in operation S910, the electronic device 1000 maydetect a surrounding vehicle from a surrounding environment image. Theelectronic device 1000 may detect a lane and the surrounding vehiclefrom the surrounding environment image, and obtain location informationabout the surrounding vehicle, by analyzing the surrounding environmentimage obtained through using the external camera 1110 (see FIGS. 1 and 2).

In an embodiment, the processor 1400 (refer to FIG. 2 ) of theelectronic device 1000 may detect the surrounding vehicle moving in thesurrounding of own vehicle by analyzing a plurality of image framesincluded in the surrounding environment image. The processor 1400 maydetect the surrounding vehicle from the surrounding environment imageusing, for example, image processing or machine learning including adeep neural network.

In operation 5920, the electronic device 1000 may measure a distancebetween the surrounding vehicle and own vehicle by using at least one ofan external radar sensor or an ultrasonic sensor. In another embodiment,the processor 1400 may obtain information about a location of thesurrounding vehicle and the distance between own vehicle and thesurrounding vehicle, by transmitting an electromagnetic wave to thesurrounding vehicle by using the radar sensor 1210 (see FIGS. 2 and 4 )mounted outside a vehicle, and receiving the electromagnetic wavereflected from the surrounding vehicle. In yet another embodiment, theprocessor 1400 may obtain information about the location of thesurrounding vehicle and the distance between own vehicle and thesurrounding vehicle, by transmitting an ultrasonic signal to thesurrounding vehicle using the ultrasonic sensor 1220 (see FIGS. 2 and 4) mounted outside the vehicle, receiving an ultrasonic echo signalreflected from the surrounding vehicle, and analyzing the receivedultrasonic echo signal.

In operation 5930, the electronic device 1000 may determine whether themeasured distance is equal to or less than a preset threshold value.

When the measured distance between own vehicle and the surroundingvehicle is equal to or less than the threshold value (operation S940),the electronic device 1000 may determine whether an expected entry routeof the surrounding vehicle is the same as an entry route of own vehicle.In yet another embodiment, the processor 1400 may track the location ofthe surrounding vehicle detected from the surrounding environment image,and may update location information about the surrounding vehicle inreal time. In yet another embodiment, the processor 1400 may predict theexpected entry route of the surrounding vehicle using the locationinformation about the surrounding vehicle detected from each of theplurality of frames included in the surrounding environment image. Theprocessor 1400 may determine whether the expected entry route of thesurrounding vehicle is the same as the entry route of own vehicle.

When it is determined that the expected entry route of the surroundingvehicle is the same as the entry route of own vehicle (operation S950),the electronic device 1000 may switch the surrounding environment imageto a top view image and may display the top view image. The processor1400 may switch the surrounding environment image displayed on the CMSside display 1310 (see FIGS. 1 and 2 ) to the top view image, and maydisplay the top view image on the CMS side display 1310.

When it is determined that the expected entry route of the surroundingvehicle is not the same as the entry route of own vehicle (operationS980), the electronic device 1000 may display the surroundingenvironment image on the CMS side display 1310. In operation S980, aview of the surrounding environment image displayed on the CMS sidedisplay 1310 from operation S810 of FIG. 8 may be continuouslymaintained.

When the measured distance between own vehicle and the surroundingvehicle exceeds the threshold value (operation S960), the electronicdevice 1000 may measure a speed of the surrounding vehicle on a lane tobe entered. In yet another embodiment, the processor 1400 may measurethe speed of the surrounding vehicle, by detecting the surroundingvehicle detected from the plurality of frames of the surroundingenvironment image and calculating a degree of movement of thesurrounding vehicle in each of the plurality of frames. In yet anotherembodiment, the processor 1400 may measure the speed of the surroundingvehicle, by calculating a distance change between the surroundingvehicle and own vehicle in real time by using at least one of theexternal radar sensor 1210 or the ultrasonic sensor 1220.

In operation S970, the electronic device 1000 may determine whether themeasured speed of the surrounding vehicle exceeds the speed of ownvehicle.

When it is determined that the measured speed of the surrounding vehicleexceeds the speed of own vehicle, the electronic device 1000 maydetermine whether the expected entry route of the surrounding vehicle isthe same as the entry route of own vehicle (operation S940).

When it is determined that the measured speed of the surrounding vehicleis less than or equal to the speed of own vehicle, the electronic device1000 may switch the surrounding environment image displayed on the CMSside display 1310 to the top view image and displays the top view image(operation S950).

FIG. 10 is a diagram illustrating an embodiment in which an electronicdevice of the disclosure detects a situation including a lane changesignal and a lane departure of a 10 and switches a view of an imagedisplayed on a CMS side display based on the detected situationaccording to an embodiment of the disclosure.

Referring to FIG. 10 , the electronic device 1000 may change the view ofthe image displayed on the CMS side display 1310 two times.

The CMS side display 1310 may display a first image that is thesurrounding environment image 100 obtained by capturing the surroundingof the vehicle 10 by using the external camera 1110 (see FIGS. 1 and 2).

When detecting a lane change signal in operation S1010, the electronicdevice 1000 may switch the first image displayed on the CMS side display1310 to a second image and may display the second image. In anembodiment, the electronic device 1000 may detect the lane change signalof the vehicle 10, by detecting a user input for manipulating the turnsignal lever 2100. In another embodiment, the electronic device 1000 maydetect the lane change signal including a turn or a lane change based onroute information of a navigation system installed in the vehicle 10.

The second image may include the top view image 110 and the surroundingenvironment image 100. In the second image, the surrounding environmentimage 100 may be overlaid on the top view image 110. In an embodiment,the electronic device 1000 may reduce a frame size of the surroundingenvironment image 100 displayed on the CMS side display 1310, mayoverlay and may display the reduced surrounding environment image 100 onthe top view image 110.

In operation S1020, the electronic device 1000 may detect a lanedeparture of the vehicle 10. In an embodiment, the electronic device1000 may detect a lane from an image by analyzing the image captured byusing the front camera 1112 disposed on the front part of the vehicle10, the rear camera 1114 disposed on the rear part of the vehicle 10,and the external camera 1110 disposed on the left and right parts of thevehicle 10. The electronic device 1000 may detect whether the vehicle 10departs from a first lane on which the vehicle 10 is currently drivingand enters a second lane which is a lane to be entered by a presetrange. For example, the electronic device 1000 may detect that thevehicle 10 has departed from the lane when the vehicle 10 enters thesecond lane which is the lane to be entered, by α % or more of a totalwidth 1 of the vehicle. For example, α % may be 40%, but is not limitedthereto.

When the lane departure is detected in operation S1020, the electronicdevice 1000 may switch the second image displayed on the CMS sidedisplay 1310 to a third image and display the third image. The thirdimage is an image including only the top view image 110 excluding thesurrounding environment image 100 overlaid on the second image.

In the embodiment shown in FIG. 10 , when the lane change signal isdetected, the electronic device 1000 of the disclosure may display thesecond image displaying the top view image 110 and the surroundingenvironment image 100 together on the CMS side display 1310, but whenthe vehicle 10 departs from the first lane on which the vehicle 10 isactually driving, the electronic device 1000 of the disclosure mayswitch the second image to the third image including only the top viewimage 110 and display the third image. Even when a driver manipulatesthe turn signal lever 2100 to light a left or right turn signal, becausethe vehicle actually continues to drive in the first lane on which thevehicle is actually driving, a probability of an accident such as acollision with the surrounding vehicle may be relatively low. In thiscase, the electronic device 1000 may provide both the top view image 110and the surrounding environment image 100. However, when the vehicle 10departs from the first lane while driving, because the driver has a highprobability of looking ahead and the risk of an accident is relativelyincreased, the electronic device 1000 may display only the top viewimage 110 that specifically displays a location relationship with thesurrounding vehicle on the CMS side display 1310. The electronic device1000 of the disclosure may detect each situation of the lane changesignal or the lane departure, and switch the view of the image displayedon the CMS side display 1310 according to the detected situation,thereby improving reliability and enhancing stability.

FIGS. 11A and 11B are diagrams illustrating an embodiment in which anelectronic device of the disclosure changes a FoV of an image displayedon a CMS side display based on a distance change between the CMS sidedisplay and a driver according to various embodiments of the disclosure.

Referring to FIG. 11A, the electronic device 1000 may obtain an occupantimage by capturing at least one of a driver or a passenger in apassenger seat by using the internal camera 1120. The internal camera1120 may be disposed, for example, on an upper end of a dashboard, aroom mirror, or a cluster inside a vehicle. However, a location at whichthe internal camera 1120 is disposed is not limited to theabove-described example.

The internal camera 1120 may capture an occupant's face to obtain theoccupant image, and provide the obtained occupant image to the processor1400 (see FIG. 2 ). The processor 1400 may obtain location informationabout the occupant including at least one of a head location of theoccupant, a head rotation direction, or a gaze direction from theoccupant image obtained from the internal camera 1120. In an embodiment,the processor 1400 may detect a driver's face or a passenger's face inthe passenger seat from the occupant image, and extract main featurepoints, such as eyes, nose, mouth, etc., from the detected face. In thiscase, the processor 1400 may use a well-known image processingtechnology or a machine learning-based image analysis technology using adeep neural network. For example, the processor 1400 may extract themain feature points of the occupant's face from the occupant image, byperforming training using a well-known neural network model, such as animage-based Convolution Neural Network (CNN) or a Recurrent NeuralNetwork (RNN). However, the disclosure is not limited thereto, and theprocessor 1400 may use all the known feature point extractiontechnologies. The processor 1400 may obtain 3D location coordinatevalues of the extracted main feature points, and obtain the occupantlocation information including at least one of the head location of theoccupant or the head rotation direction using the obtained the 3Dlocation coordinate values.

In an embodiment, the internal camera 1120 may include an eye trackingsensor that tracks the gaze of the occupant by capturing occupant'spupils including the driver or the passenger in the passenger seat, anddetecting a movement of the pupils. The processor 1400 may obtaininformation about the gaze direction of the occupant from the eyetracking sensor included in the internal camera 1120.

The electronic device 1000 may measure a distance between the occupant'sface and the CMS side display 1310 using the 3D location coordinatevalues of the main feature points extracted from the occupant image anda location coordinate value of the CMS side display 1310. In theembodiment shown in FIG. 11A, the electronic device 1000 may measure afirst distance d₁ between the occupant's face and the CMS side display1310.

The electronic device 1000 may adjust the FoV of the top view image 110displayed on the CMS side display 1310 based on the measured distancebetween the occupant's face and the CMS side display 1310. In theembodiment shown in FIG. 11A, when the distance between the occupant'sface and the CMS side display 1310 is the first distance d₁, the FoV ofthe top view image 110 may include only the own vehicle image 121 andthe first surrounding vehicle image 122.

Referring to FIG. 11B, the occupant (e.g., a driver) may approach theoccupant's face in a direction adjacent to the CMS side display 1310 soas to take a closer look at the CMS side display 1310. In this case, theinternal camera 1120 may capture the occupant's face in real time, andprovide the occupant image obtained through capturing to the processor1400. The processor 1400 may extract the main feature points of theoccupant's face by analyzing the occupant image, obtain the 3D locationcoordinate values of the extracted main feature points, and measure thechanged distance between the occupant's face and the CMS side display1310 using the obtained 3D location coordinate values of the mainfeature points and the location coordinate value of the CMS side display1310. In the embodiment shown in FIG. 11B, the distance between adriver's face and the CMS side display 1310 may be measured as a seconddistance d₂. A value of the second distance d₂ may be smaller than thefirst distance d₁ (see FIG. 11A).

The electronic device 1000 may zoom in or out the FoV of a top viewimage 112 based on a change in the distance value between the occupant'sface and the CMS side display 1310. In the embodiment shown in FIG. 11B,when the distance between the driver's face and the CMS side display1310 is the second distance d₂, the top view image 112 displayed on theCMS side display 1310 may display not only the own vehicle image 121 andthe first surrounding vehicle image 122 but also a second surroundingvehicle image 126. Referring to FIG. 11A together, when the distancebetween the occupant (driver)'s face and the CMS side display 1310 ischanged from the first distance di to the second distance d₂, that is,when the occupant (driver) moves the face adjacent to the CMS sidedisplay 1310 so as to take a closer look at the CMS side display 1310,the electronic device 1000 may zoom out the top view image 112 displayedon the CMS side display 1310. The top view image 112 shown in FIG. 11Bis a zoomed-out image compared to the top view image 110 shown in FIG.11A, and therefore, the own vehicle image 121 and the first surroundingvehicle image 122 may be displayed on the top view image 112 in asmaller scale than that shown in FIG. 11A.

In the opposite embodiment, that is, when the occupant (driver) movesthe face from the CMS side display 1310 to a headrest direction of avehicle seat, the distance between the occupant (driver)'s face and theCMS side display 1310 may be greater than the second distance d₂, andthe electronic device 1000 may zoom in the top view image 112.

FIG. 12 is a diagram illustrating an embodiment in which an electronicdevice of the disclosure changes a FoV of an image displayed on a CMSside display based on a hand gesture of an occupant in a vehicleaccording to an embodiment of the disclosure.

Referring to FIG. 12 , the top view image 110 may be displayed on theCMS side display 1310. The own vehicle image 121 and the firstsurrounding vehicle image 122 may be displayed on the top view image110.

The electronic device 1000 may detect the hand gesture of the occupant.In an embodiment, the hand gesture input unit 1720 (see FIG. 2 ) of theelectronic device 1000 may receive the hand gesture of a driver or apassenger in a passenger seat including at least one of a pinch in/pinchout or a palm swipe. The processor 1400 (see FIG. 2 ) may zoom in/outthe FoV of the top view image 112 displayed on the CMS side display 1310based on a hand gesture input received through the hand gesture inputunit 1720. For example, when detecting a pinch out input of pinchingboth fingers, the electronic device 1000 may zoom out the top view image112 displayed on the CMS side display 1310. In the embodiment shown inFIG. 12 , when the top view image 112 is zoomed out, not only the ownvehicle image 121 and the first surrounding vehicle image 122 but alsothe second surrounding vehicle image 126 may be displayed on the topview image 112.

The top view image 112 is a zoomed-out image compared to the top viewimage 110 displayed on the CMS side display 1310 before the hand gestureis detected, and therefore, the own vehicle image 121 and the firstsurrounding vehicle image 122 may be displayed on the top view image 112in a smaller scale than that of the top view image 110 before the handgesture is detected.

To the contrary, for example, when detecting a pinch-in input ofspreading both fingers, the electronic device 1000 may zoon in the topview image 112 displayed on the CMS side display 1310.

For example, the electronic device 1000 may zoom in/out the top viewimage 112 displayed on the CMS side display 1310 according to adirection of a palm swipe input.

In the embodiments of FIGS. 11A, 11B, and 12 , the electronic device1000 may build an occupant monitoring system (OMS) that detects anaction of not only the driver but also the passenger, for example, anaction of moving the face adjacent to or away from the CMS side display1310, or a hand gesture action, and may automatically adjusts the FoV ofthe top view image 110 displayed on the CMS side display 1310 based onthe detected action. The electronic device 1000 of the disclosure mayautomatically detect needs of the occupant and may provide a relateduser experience (UX), and thus, the driver may not need to perform aconscious and additional action of manipulating the CMS side display1310 while driving, and may concentrate on the driving itself, therebyimproving convenience. In addition, the driver may omit unnecessarymanipulation of the CMS side display 1310 or button manipulation,thereby improving driving stability.

FIG. 13 is a diagram illustrating an embodiment in which an electronicdevice of the disclosure displays a top view image on a CID according toan embodiment of the disclosure.

Referring to FIG. 13 , the CID 1320 may display the top view image 110and an image content 150. In an embodiment, the CID 1320 may split intoa first region 1322 and a second region 1324, the image content 150 maybe displayed on the first region 1322, and the top view image 110 may bedisplayed on the second region 1324. The image content 150 may include,for example, at least one of a movie, a television (TV) show,Internet-based video streaming (e.g., YouTube or Netflix), or a game.

In the embodiment shown in FIG. 13 , when a vehicle is driving in anautonomous driving mode, the CID 1320 may display the image content 150that an occupant may enjoy and simultaneously display the top view image110 on a CMS side display together, so that a driver may predict adetermination of the vehicle itself and an autonomous driving method ina lane change or turn situation, etc., thereby providing highreliability regarding autonomous driving to the driver or a passenger.

FIG. 14A is a diagram illustrating an embodiment in which an electronicdevice of the disclosure displays a top view image on a CMS side displayand a CID in the case of left-hand drive (LHD) vehicle according to anembodiment of the disclosure.

FIG. 14B is a diagram illustrating an embodiment in which an electronicdevice of the disclosure displays a top view image on a CMS side displayand a CID 1320 in the case of a right-hand drive (RHD) vehicle accordingto an embodiment of the disclosure.

Types of vehicles may be classified into the LHD vehicle and the RHDvehicle according to a location of a steering wheel. The type of vehiclemay vary depending on a road. For example, in a country (e.g., S. Koreaor the United States) that has adopted a traffic system in whichvehicles run on the right side of the road, the LHD vehicle with thesteering wheel located on the left is used, and in a country (e.g.,Japan or UK) that has adopted a traffic system in which vehicles run onthe left side of the road, the RHD vehicle with the steering wheellocated on the right is used.

Referring to FIG. 14A, when the LHD vehicle turns a driving direction tothe right or attempts to change a lane to a right lane with respect tothe vehicle, the vehicle may receive an input activating (lighting) aright turn signal by manipulating the turn signal lever 2100 by thedriver. When the right turn signal is activated, the electronic device1000 may receive turn signal activation information from the vehiclesensor module 2000 (see FIG. 3 ), and may switch a surroundingenvironment image displayed on the right CMS side display 1310R to thetop view image 110 based on the received turn signal activationinformation. In this case, the electronic device 1000 may display thetop view image 110 not only on the right CMS side display 1310R but alsoon the CID 1320.

When the driver manipulates (lights) a left turn signal by manipulatingthe turn signal lever 2100, the electronic device 1000 may switch thesurrounding environment image 100 displayed on the left CMS side display1310L to the top view image 110. However, in this case, the top viewimage 110 may not be displayed on the CID 1320.

Referring to FIG. 14B, when the RHD vehicle turns the driving directionto the left or attempts to change a lane to a left lane with respect tothe vehicle, the vehicle may receive an input activating (lighting) aleft turn signal by manipulating the turn signal lever 2100 by thedriver. When the left turn signal is activated, the electronic device1000 may receive turn signal activation information from the vehiclesensor module 2000 (see FIG. 3 ), and may switch a surroundingenvironment image displayed on the left CMS side display 1310L to thetop view image 110 based on the received turn signal activationinformation. In this case, the electronic device 1000 may display thetop view image 110 not only on the left CMS side display 1310L but alsoon the CID 1320.

When the driver manipulates (lights) the right turn signal bymanipulating the turn signal lever 2100, the electronic device 1000 mayswitch the surrounding environment image 100 displayed on the right CMSside display 1310R to the top view image 110. However, in this case, thetop view image 110 may not be displayed on the CID 1320.

In the embodiments shown in FIGS. 14A and 14B, the CMS side displaylocated adjacent to the driver among the left CMS side display 1310L andthe right CMS side display 1310R is different according to whether thevehicle type is the LHD vehicle or the RHD vehicle. For example, in thecase of the LHD vehicle, the left CMS side display 1310L is disposedadjacent to a location of the driver, but the right CMS side display1310R is disposed relatively far from the location of the driver.Accordingly, the driver needs an additional action to rotate the face orturn a gaze direction to the right to view the right CMS side display1310R, and in this case, concentration may be dispersed. Although itdoes not matter while the driver stops the vehicle, but there is a riskof an accident occurring when concentration is lowered to view the rightCMS side display 1310R while driving or when changing lanes. In the caseof the LHD vehicle, when the right turn signal is activated, theelectronic device 1000 may display the top view image 110 not only onthe CID 1320 but also the right CMS side display 1310R, and therefore,the additional action of the driver may be omitted, thereby improvingdriving stability. In the case of the RHD vehicle, only the direction ofthe turn signal is different, but the principle of operation is thesame.

A program executed by the electronic device 1000 described in thespecification may be implemented by hardware components, softwarecomponents, and/or a combination of hardware components and softwarecomponents. The program may be performed by all systems capable ofperforming computer-readable instructions.

Software may include a computer program, code, instructions, or acombination thereof, and constitute a processing device so as to operateas desired, or independently or collectively command the processingdevice.

The software may be implemented by a computer program includinginstructions stored in computer-readable storage media. Examples of thecomputer-readable storage media may include magnetic storage media(e.g., ROM, RAM, floppy disks, hard disks, etc.) and optical recordingmedia (e.g., CD-ROMs, digital versatile discs (DVDs), etc.). Thecomputer-readable storage media may be distributed over network coupledcomputer systems so that the computer-readable code is stored andexecuted in a distributed fashion. The media may be read by a computer,stored in a memory, and executed by a processor.

The computer-readable storage media may be provided in the form ofnon-transitory storage media. Herein, the term ‘non-transitory’ merelyindicates that a storage medium is tangible without including a signal,and does not discriminate whether data is semipermanently or temporarilystored in a storage medium.

In addition, the program according to the embodiments of thespecification may be provided by being included in a computer programproduct. The computer program product may be traded between a seller anda purchaser.

The computer program product may include a software program or anon-transitory computer-readable storage medium in which the softwareprogram is stored. For example, the computer program product may includea software program form of product (e.g., a downloadable application)electronically distributed through a manufacturing company of theelectronic device or an electronic market (e.g., Google PlayStore™, orApp Store™). For the electronic distribution, at least a portion of thesoftware program may be stored in a storage medium or temporarilygenerated. In this case, the storage medium may be included in a serverof the manufacturing company of a vehicle or the electronic device 1000,a server of the electronic market, or a proxy server configured totemporarily store the software program.

The computer program product may include a storage medium of a server ora storage medium of a device in a system including the electronic device1000, the server and the other device. Alternatively, when a thirddevice (e.g., a smartphone) communicatively connected to the electronicdevice 1000 exists, the computer program product may include a storagemedium of the third device. Alternatively, the computer program productmay include the software program to be transmitted from the electronicdevice 1000 to the device or the third device or transmitted from thethird device to the device.

In this case, one of the electronic device 1000, the device, and thethird device may execute the computer program product and perform themethods according to the embodiments of the disclosure. Alternatively,two or more of the electronic device 1000, the device, and the thirddevice may execute the computer program product and perform the methodsaccording to the embodiments of the disclosure in a distributed fashion.

For example, the electronic device 1000 may execute the computer programproduct stored in the memory (1500 see FIG. 2 ) to control the otherdevice communicatively connected to the electronic device 1000 toperform the methods according to the embodiments of the disclosure.

For another example, the third device may execute the computer programproduct to control a device communicatively connected to the thirddevice to perform the method according to the embodiment of thedisclosure.

When the third device executes the computer program product, the thirddevice may download the computer program product from the electronicdevice 1000 and execute the downloaded computer program product.Alternatively, the third device may execute the computer program productprovided in a pre-loaded state to perform the methods according to theembodiments of the disclosure.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those of skilledin the art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. A method, performed by an electronic deviceinstalled in a vehicle, of displaying an image by using a cameramonitoring system (CMS) of the vehicle, the method comprising:displaying, on a CMS side display, a first image that is a surroundingenvironment image obtained by capturing a left surrounding environmentor a right surrounding environment of the vehicle by using an externalcamera system disposed on a side surface of the vehicle; detecting alane change signal of the vehicle; in response to the detected lanechange signal, switching the first image displayed on the CMS sidedisplay to a second image that is a top view image showing locations ofthe vehicle and a surrounding vehicle in a virtual image as looking downfrom above the vehicle, and displaying the second image; and displaying,on the second image, a lane change user interface (UI) indicatinginformation about whether a lane change is possible.
 2. The method ofclaim 1, wherein the detecting of the lane change signal comprisesdetecting a user input for lighting a turn signal, by the user inputcomprising a manipulation of a turn signal lever of the vehicle.
 3. Themethod of claim 1, wherein the detecting of the lane change signalcomprises detecting one of a turn or a lane change based on drivingroute information of a navigation system of the vehicle.
 4. The methodof claim 1, wherein the displaying of the lane change UI indicatinginformation about whether the lane change is possible comprisesdisplaying a UI related to driving environment information comprising atleast one of a lane, a location of a surrounding vehicle, a relativespeed between the vehicle and the surrounding vehicle, a distancebetween the vehicle and the surrounding vehicle, or an expected entryroute of each of the vehicle and the surrounding vehicle detected fromthe surrounding environment image.
 5. The method of claim 1, wherein thedisplaying of the second image comprises overlaying and displaying thesurrounding environment image obtained through the external camerasystem on the top view image.
 6. The method of claim 5, furthercomprising: detecting a lane departure based on the vehicle departingfrom a first lane on which the vehicle is currently driving and enteringa second lane which is a lane to be entered by a preset range; inresponse to the detected lane departure, switching the second imagedisplayed on the CMS side display to a third image comprising only a topview image excluding the overlaid surrounding environment image in thesecond image; and displaying the third image on the CMS side display. 7.The method of claim 1, further comprising: obtaining an occupant imageby using a camera mounted inside the vehicle; obtaining locationinformation about an occupant, comprising at least one of a headlocation, a head rotation direction, or a gaze direction of the occupantfrom the obtained occupant image; measuring a distance between theoccupant and the CMS side display based on the obtained locationinformation of the occupant; and zooming in or out a field of view (FoV)of the second image based on the measured distance.
 8. An electronicdevice configured to display an image using a camera monitoring system(CMS) of a vehicle, the electronic device comprising: a first externalcamera disposed on a left side surface of an outside of the vehicle anda second external camera disposed on a right side surface of the outsideof the vehicle, wherein the first external camera and the secondexternal camera are configured to: capture a surrounding environment ofthe vehicle, and obtain a first image that is a surrounding environmentimage; a CMS side display disposed inside the vehicle displaying thesurrounding environment image; a memory storing a program comprising oneor more instructions controlling the electronic device; and at least oneprocessor configured to execute the one or more instructions of theprogram stored in the memory, wherein the processor is furtherconfigured to: detect a lane change signal of the vehicle, in responseto the detected lane change signal: switch the first image displayed onthe CMS side display to a second image that is a top view image showinglocations of the vehicle and a surrounding vehicle in a virtual image aslooking down from above the vehicle, and control the CMS side display todisplay the second image, and display a lane change user interface (UI)indicating information about whether a lane change is possible on thesecond image.
 9. The electronic device of claim 8, wherein the processoris further configured to: detect a user input for lighting a turnsignal, the user input comprising a manipulation of a turn signal leverof the vehicle, and switch the first image displayed on the CMS sidedisplay to the second image based on the user input.
 10. The electronicdevice of claim 8, wherein the processor is further configured todisplay a UI related to driving environment information comprising atleast one of a lane, a location of a surrounding vehicle, a relativespeed between the vehicle and the surrounding vehicle, a distancebetween the vehicle and the surrounding vehicle, or an expected entryroute of each of the vehicle and the surrounding vehicle detected fromthe surrounding environment image on the CMS side display.
 11. Theelectronic device of claim 8, wherein the processor is furtherconfigured to: control the CMS side display to overlay, and display thesurrounding environment image obtained through the first external cameraand the second external camera on the top view image.
 12. The electronicdevice of claim 11, wherein the processor is further configured todetect a lane departure that the vehicle departs from a first lane onwhich the vehicle is currently driving and enters a second lane which isa lane to be entered by a preset range, and wherein in response to thedetected lane departure: switch the second image displayed on the CMSside display to a third image comprising only a top view image excludingthe overlaid surrounding environment image, and control the CMS sidedisplay to display the third image.
 13. The electronic device of claim8, further comprising: an internal camera mounted inside the vehicle andconfigured to obtain an occupant image by capturing an occupant, whereinthe processor is further configured to: obtain location information ofthe occupant comprising at least one of a head location, a head rotationdirection, or a gaze direction of the occupant from the occupant image,measure a distance between a driver and the CMS side display based onthe obtained location information of the occupant, and zoom in or out afield of view (FoV) of the second image based on the measured distance.14. The electronic device of claim 8, further comprising: a centerinformation display (CID) disposed on a dashboard of the vehicle,wherein the processor is further configured to display the second imageon the CMS side display and the CID.
 15. A non-transitorycomputer-readable recording medium recording thereon at least oneprogram for implementing the method according to claim 1.